Perception methods and systems for low lighting conditions

ABSTRACT

Methods and systems are provided for detecting objects within an environment of a vehicle. In one embodiment, a method includes: receiving, by a processor, image data sensed from the environment of the vehicle; determining, by a processor, an area within the image data that object identification is uncertain; controlling, by the processor, a position of a lighting device to illuminate a location in the environment of the vehicle, wherein the location is associated with the area; controlling, by the processor, a position of one or more sensors to obtain sensor data from the location of the environment of the vehicle while the lighting device is illuminating the location; identifying, by the processor, one or more objects from the sensor data; and controlling, by the processor, the vehicle based on the one or more objects.

INTRODUCTION

The present disclosure generally relates to autonomous vehicles, andmore particularly relates to systems and methods for improvingperception of an environment of an autonomous vehicle during lowlighting conditions.

An autonomous vehicle is a vehicle that is capable of sensing itsenvironment and navigating with little or no user input. For example, anautonomous vehicle may include a perception system that senses itsenvironment using sensing devices such as radar, lidar, image sensors,and the like and, based on the sensed data, observes objects and otheractivity within the environment. The autonomous vehicle system uses thisinformation from the perception system and information from othersystems such as global positioning systems (GPS) technology, navigationsystems, vehicle-to-vehicle communication, vehicle-to-infrastructuretechnology, and/or drive-by-wire systems to navigate the vehicle.

In some instance, the perception system has difficulty observing theenvironment. For example, during low lighting conditions (e.g., atnight, enclosed areas, etc.) identifying objects can be difficult; evenwith the help of active sensors such as Lidar because of theirrelatively low resolution.

Accordingly, it is desirable to provide systems and methods that provideimproved perception of the vehicle environment. Furthermore, otherdesirable features and characteristics of the present disclosure willbecome apparent from the subsequent detailed description and theappended claims, taken in conjunction with the accompanying drawings andthe foregoing technical field and background.

SUMMARY

Methods and systems are provided for detecting objects within anenvironment of a vehicle. In one embodiment, a method includes:receiving, by a processor, image data sensed from the environment of thevehicle; determining, by a processor, an area within the image data thatobject identification is uncertain; controlling, by the processor, aposition of a lighting device to illuminate a location in theenvironment of the vehicle, wherein the location is associated with thearea; controlling, by the processor, a position of one or more sensorsto obtain sensor data from the location of the environment of thevehicle while the lighting device is illuminating the location;identifying, by the processor, one or more objects from the sensor data;and controlling, by the processor, the vehicle based on the one or moreobjects.

In various embodiments, the one or more sensors comprise a camera and alidar or ridar.

In various embodiments, the position of the lighting device includes atleast one of a yaw, and a pitch relative to the vehicle.

In various embodiments, the position of the one or more sensors includesat least one of a yaw, a pitch, and a roll relative to the vehicle.

In various embodiments, the method includes determining a priority ofthe area, and wherein the controlling the position of the lightingdevice and the controlling the position of the one or more sensors isbased on the priority of the area.

In various embodiments, the determining the priority is based on lidardata sensed from the environment of the vehicle.

In various embodiments, the determining the priority is based on alocation of the area relative to lane of travel of the vehicle.

In various embodiments, the determining the priority is based on adetected potential object in the area.

In various embodiments, the determining the area within the image thatobject detection is uncertain is based on a pixel value for thresholdamount of pixels within the image.

In another embodiment, a computer implemented system for detectingobjects within an environment of a vehicle, includes: a computerreadable medium configured to store instructions; and a processorconfigured to perform the instructions, the instructions when performed,receive image data sensed from the environment of the vehicle, determinean area within the image data that object identification is uncertain,control a position of a lighting device to illuminate a location in theenvironment of the vehicle, wherein the location is associated with thearea, control a position of one or more sensors to obtain sensor datafrom the location of the environment of the vehicle while the lightingdevice is illuminating the location, identify one or more objects fromthe sensor data, and control the vehicle based on the one or moreobjects.

In various embodiments, the one or more sensors comprise a camera and alidar or a radar.

In various embodiments, the position of the lighting device includes atleast one of a yaw, and a pitch relative to the vehicle.

In various embodiments, the position of the one or more sensors includesat least one of a yaw, a pitch, and a roll relative to the vehicle.

In various embodiments, the instructions determine a priority of thearea, and wherein the controlling the position of the lighting deviceand the controlling the position of the one or more sensors is based onthe priority of the area.

In various embodiments, the instructions determine the priority based onlidar data sensed from the environment of the vehicle.

In various embodiments, the instructions determine the priority based ona location of the area relative to lane of travel of the vehicle.

In various embodiments, the instructions determine the priority based ona detected potential object in the area.

In various embodiments, the instructions determine the area within theimage that object detection is uncertain is based on a pixel value forthreshold amount of pixels within the image.

In another embodiment, an autonomous vehicle includes: a camera thatsenses image data from an environment of the autonomous vehicle; atleast one of a lidar and a radar that senses sensor data from theenvironment of the vehicle; a steerable lighting device that selectivelyilluminates the environment of the vehicle; and a controller configuredto, by a processor, receive image data sensed from the environment ofthe vehicle, determine an area within the image data that objectidentification is uncertain, control a position of the lighting deviceto illuminate a location in the environment of the vehicle, wherein thelocation is associated with the area, control a position of the cameraand the at least one of lidar and radar to obtain the image data and thesensor data from the location of the environment of the vehicle whilethe lighting device is illuminating the location, identify one or moreobjects from the sensor data, and control the vehicle based on the oneor more objects.

In various embodiments, the camera, the at least one of lidar and radar,and the lighting device are movably coupled to the vehicle, and whereinthe position is relative to the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary embodiments will hereinafter be described in conjunctionwith the following drawing figures, wherein like numerals denote likeelements, and wherein:

FIG. 1 is a functional block diagram illustrating an autonomous vehiclehaving a perception system, in accordance with various embodiments;

FIG. 2 is a functional block diagram illustrating a transportationsystem having one or more autonomous vehicles of FIG. 1, in accordancewith various embodiments;

FIGS. 3 and 4 are dataflow diagrams illustrating an autonomous drivingsystem that includes the perception system of the autonomous vehicle, inaccordance with various embodiments; and

FIG. 5 is a flowchart illustrating a control method for controlling theautonomous vehicle, in accordance with various embodiments.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the application and uses. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe preceding technical field, background, brief summary or thefollowing detailed description. As used herein, the term module refersto any hardware, software, firmware, electronic control component,processing logic, and/or processor device, individually or in anycombination, including without limitation: application specificintegrated circuit (ASIC), an electronic circuit, a processor (shared,dedicated, or group) and memory that executes one or more software orfirmware programs, a combinational logic circuit, and/or other suitablecomponents that provide the described functionality.

Embodiments of the present disclosure may be described herein in termsof functional and/or logical block components and various processingsteps. It should be appreciated that such block components may berealized by any number of hardware, software, and/or firmware componentsconfigured to perform the specified functions. For example, anembodiment of the present disclosure may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices. In addition, those skilled inthe art will appreciate that embodiments of the present disclosure maybe practiced in conjunction with any number of systems, and that thesystems described herein is merely exemplary embodiments of the presentdisclosure.

For the sake of brevity, conventional techniques related to signalprocessing, data transmission, signaling, control, and other functionalaspects of the systems (and the individual operating components of thesystems) may not be described in detail herein. Furthermore, theconnecting lines shown in the various figures contained herein areintended to represent example functional relationships and/or physicalcouplings between the various elements. It should be noted that manyalternative or additional functional relationships or physicalconnections may be present in an embodiment of the present disclosure.

With reference to FIG. 1, a perception system in accordance with thepresent disclosure is shown generally at 100 and is associated with avehicle 10 in accordance with various embodiments. In general, theperception system 100 generates an attention signal, to steerspot-lights, head-lights, or other exterior lighting of the vehicle 10to identified areas of the vehicle's environment, to illuminate low litareas and improve the resulting observations sensors directed to thoseareas of the environment. The vehicle 10 intelligently controls thevehicle 10 based on the improved resulting observations.

As depicted in FIG. 1, the vehicle 10 generally includes a chassis 12, abody 14, front wheels 16, and rear wheels 18. The body 14 is arranged onthe chassis 12 and substantially encloses components of the vehicle 10.The body 14 and the chassis 12 may jointly form a frame. The wheels16-18 are each rotationally coupled to the chassis 12 near a respectivecorner of the body 14.

In various embodiments, the vehicle 10 is an autonomous vehicle and theperception system 100 is incorporated into the autonomous vehicle 10(hereinafter referred to as the autonomous vehicle 10). The autonomousvehicle 10 is, for example, a vehicle that is automatically controlledto carry passengers from one location to another. The vehicle 10 isdepicted in the illustrated embodiment as a passenger car, but it shouldbe appreciated that any other vehicle including motorcycles, trucks,sport utility vehicles (SUVs), recreational vehicles (RVs), marinevessels, aircraft, etc., can also be used. In an exemplary embodiment,the autonomous vehicle 10 is a so-called Level Four or Level Fiveautomation system. A Level Four system indicates “high automation”,referring to the driving mode-specific performance by an automateddriving system of all aspects of the dynamic driving task, even if ahuman driver does not respond appropriately to a request to intervene. ALevel Five system indicates “full automation”, referring to thefull-time performance by an automated driving system of all aspects ofthe dynamic driving task under all roadway and environmental conditionsthat can be managed by a human driver.

As shown, the autonomous vehicle 10 generally includes a propulsionsystem 20, a transmission system 22, a steering system 24, a brakesystem 26, a sensor system 28, an actuator system 30, at least one datastorage device 32, at least one controller 34, and a communicationsystem 36. The propulsion system 20 may, in various embodiments, includean internal combustion engine, an electric machine such as a tractionmotor, and/or a fuel cell propulsion system. The transmission system 22is configured to transmit power from the propulsion system 20 to thevehicle wheels 16-18 according to selectable speed ratios. According tovarious embodiments, the transmission system 22 may include a step-ratioautomatic transmission, a continuously-variable transmission, or otherappropriate transmission. The brake system 26 is configured to providebraking torque to the vehicle wheels 16-18. The brake system 26 may, invarious embodiments, include friction brakes, brake by wire, aregenerative braking system such as an electric machine, and/or otherappropriate braking systems. The steering system 24 influences aposition of the of the vehicle wheels 16-18. While depicted as includinga steering wheel for illustrative purposes, in some embodimentscontemplated within the scope of the present disclosure, the steeringsystem 24 may not include a steering wheel.

The sensor system 28 includes one or more sensing devices 40 a-40 n thatsense observable conditions of the exterior environment and/or theinterior environment of the autonomous vehicle 10. The sensing devices40 a-40 n can include, but are not limited to, radars, lidars, globalpositioning systems, optical cameras, thermal cameras, ultrasonicsensors, inertial measurement units, and/or other sensors. In variousembodiments, the sensing devices 40 a-40 n can include one or moreoptical cameras, high resolution lidar sensors, and low resolution lidarsensors.

The actuator system 30 includes one or more actuator devices 42 a-42 nthat control one or more vehicle features such as, but not limited to,the propulsion system 20, the transmission system 22, the steeringsystem 24, and the brake system 26. In various embodiments, the vehiclefeatures can further include interior and/or exterior vehicle featuressuch as, but are not limited to, doors, a trunk, and cabin features suchas air, music, lighting, etc. (not numbered). In various embodiments,the vehicle features include exterior lighting devices 27 such as, butnot limited to, headlights or spotlights that project light towards theenvironment of the vehicle 10. In various embodiments, the vehiclefeatures can include positioning devices that movably couple one or moreof the sensing devices 40 a-40 n, and/or the lighting devices 27 and toadjust a pitch, a yaw, and/or a roll of the associated sensing device 40a-40 n and/or the lighting devices 27 relative to the vehicle 10.

The communication system 36 is configured to wirelessly communicateinformation to and from other entities 48, such as but not limited to,other vehicles (“V2V” communication,) infrastructure (“V2I”communication), remote systems, and/or personal devices (described inmore detail with regard to FIG. 2). In an exemplary embodiment, thecommunication system 36 is a wireless communication system configured tocommunicate via a wireless local area network (WLAN) using IEEE 802.11standards or by using cellular data communication. However, additionalor alternate communication methods, such as a dedicated short-rangecommunications (DSRC) channel, are also considered within the scope ofthe present disclosure. DSRC channels refer to one-way or two-wayshort-range to medium-range wireless communication channels specificallydesigned for automotive use and a corresponding set of protocols andstandards.

The data storage device 32 stores data for use in automaticallycontrolling the autonomous vehicle 10. In various embodiments, the datastorage device 32 stores defined maps of the navigable environment. Invarious embodiments, the defined maps may be predefined by and obtainedfrom a remote system (described in further detail with regard to FIG.2). For example, the defined maps may be assembled by the remote systemand communicated to the autonomous vehicle 10 (wirelessly and/or in awired manner) and stored in the data storage device 32. As can beappreciated, the data storage device 32 may be part of the controller34, separate from the controller 34, or part of the controller 34 andpart of a separate system.

The controller 34 includes at least one processor 44 and a computerreadable storage device or media 46. The processor 44 can be any custommade or commercially available processor, a central processing unit(CPU), a graphics processing unit (GPU), an auxiliary processor amongseveral processors associated with the controller 34, a semiconductorbased microprocessor (in the form of a microchip or chip set), amacroprocessor, any combination thereof, or generally any device forexecuting instructions. The computer readable storage device or media 46may include volatile and nonvolatile storage in read-only memory (ROM),random-access memory (RAM), and keep-alive memory (KAM), for example.KAM is a persistent or non-volatile memory that may be used to storevarious operating variables while the processor 44 is powered down. Thecomputer-readable storage device or media 46 may be implemented usingany of a number of known memory devices such as PROMs (programmableread-only memory), EPROMs (electrically PROM), EEPROMs (electricallyerasable PROM), flash memory, or any other electric, magnetic, optical,or combination memory devices capable of storing data, some of whichrepresent executable instructions, used by the controller 34 incontrolling the autonomous vehicle 10.

The instructions may include one or more separate programs, each ofwhich comprises an ordered listing of executable instructions forimplementing logical functions. The instructions, when executed by theprocessor 44, receive and process signals from the sensor system 28,perform logic, calculations, methods and/or algorithms for automaticallycontrolling the components of the autonomous vehicle 10, and generatecontrol signals to the actuator system 30 to automatically control thecomponents of the autonomous vehicle 10 based on the logic,calculations, methods, and/or algorithms. Although only one controller34 is shown in FIG. 1, embodiments of the autonomous vehicle 10 caninclude any number of controllers 34 that communicate over any suitablecommunication medium or a combination of communication mediums and thatcooperate to process the sensor signals, perform logic, calculations,methods, and/or algorithms, and generate control signals toautomatically control features of the autonomous vehicle 10.

In various embodiments, one or more instructions of the controller 34are embodied in the perception system 100 and, when executed by theprocessor 44, generate an attention signal, to steer, via one or more ofthe actuator devices 42 a-42 n, the exterior lighting devices 27 of thevehicle 10 and/or one or more sensing devices 40 a-40 n to an identifiedarea. The instructions further generate signals to illuminate thelighting devices 27 towards low lit areas or other uncertain areas andimprove the resulting observations by the sensing devices 40 a-40 ndirected towards the area.

With reference now to FIG. 2, in various embodiments, the autonomousvehicle 10 described with regard to FIG. 1 may be suitable for use inthe context of a taxi or shuttle system in a certain geographical area(e.g., a city, a school or business campus, a shopping center, anamusement park, an event center, or the like) or may simply be managedby a remote system. For example, the autonomous vehicle 10 may beassociated with an autonomous vehicle based remote transportationsystem. FIG. 2 illustrates an exemplary embodiment of an operatingenvironment shown generally at 50 that includes an autonomous vehiclebased remote transportation system 52 that is associated with one ormore autonomous vehicles 10 a-10 n as described with regard to FIG. 1.In various embodiments, the operating environment 50 further includesone or more user devices 54 that communicate with the autonomous vehicle10 and/or the remote transportation system 52 via a communicationnetwork 56.

The communication network 56 supports communication as needed betweendevices, systems, and components supported by the operating environment50 (e.g., via tangible communication links and/or wireless communicationlinks). For example, the communication network 56 can include a wirelesscarrier system 60 such as a cellular telephone system that includes aplurality of cell towers (not shown), one or more mobile switchingcenters (MSCs) (not shown), as well as any other networking componentsrequired to connect the wireless carrier system 60 with a landcommunications system. Each cell tower includes sending and receivingantennas and a base station, with the base stations from different celltowers being connected to the MSC either directly or via intermediaryequipment such as a base station controller. The wireless carrier system60 can implement any suitable communications technology, including forexample, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g.,4G LTE or 5G LTE), GSM/GPRS, or other current or emerging wirelesstechnologies. Other cell tower/base station/MSC arrangements arepossible and could be used with the wireless carrier system 60. Forexample, the base station and cell tower could be co-located at the samesite or they could be remotely located from one another, each basestation could be responsible for a single cell tower or a single basestation could service various cell towers, or various base stationscould be coupled to a single MSC, to name but a few of the possiblearrangements.

Apart from including the wireless carrier system 60, a second wirelesscarrier system in the form of a satellite communication system 64 can beincluded to provide uni-directional or bi-directional communication withthe autonomous vehicles 10 a-10 n. This can be done using one or morecommunication satellites (not shown) and an uplink transmitting station(not shown). Uni-directional communication can include, for example,satellite radio services, wherein programming content (news, music,etc.) is received by the transmitting station, packaged for upload, andthen sent to the satellite, which broadcasts the programming tosubscribers. Bi-directional communication can include, for example,satellite telephony services using the satellite to relay telephonecommunications between the vehicle 10 and the station. The satellitetelephony can be utilized either in addition to or in lieu of thewireless carrier system 60.

A land communication system 62 may further be included that is aconventional land-based telecommunications network connected to one ormore landline telephones and connects the wireless carrier system 60 tothe remote transportation system 52. For example, the land communicationsystem 62 may include a public switched telephone network (PSTN) such asthat used to provide hardwired telephony, packet-switched datacommunications, and the Internet infrastructure. One or more segments ofthe land communication system 62 can be implemented through the use of astandard wired network, a fiber or other optical network, a cablenetwork, power lines, other wireless networks such as wireless localarea networks (WLANs), or networks providing broadband wireless access(BWA), or any combination thereof. Furthermore, the remotetransportation system 52 need not be connected via the landcommunication system 62, but can include wireless telephony equipment sothat it can communicate directly with a wireless network, such as thewireless carrier system 60.

Although only one user device 54 is shown in FIG. 2, embodiments of theoperating environment 50 can support any number of user devices 54,including multiple user devices 54 owned, operated, or otherwise used byone person. Each user device 54 supported by the operating environment50 may be implemented using any suitable hardware platform. In thisregard, the user device 54 can be realized in any common form factorincluding, but not limited to: a desktop computer; a mobile computer(e.g., a tablet computer, a laptop computer, or a netbook computer); asmartphone; a video game device; a digital media player; a piece of homeentertainment equipment; a digital camera or video camera; a wearablecomputing device (e.g., smart watch, smart glasses, smart clothing); orthe like. Each user device 54 supported by the operating environment 50is realized as a computer-implemented or computer-based device havingthe hardware, software, firmware, and/or processing logic needed tocarry out the various techniques and methodologies described herein. Forexample, the user device 54 includes a microprocessor in the form of aprogrammable device that includes one or more instructions stored in aninternal memory structure and applied to receive binary input to createbinary output. In some embodiments, the user device 54 includes a GPSmodule capable of receiving GPS satellite signals and generating GPScoordinates based on those signals. In other embodiments, the userdevice 54 includes cellular communications functionality such that thedevice carries out voice and/or data communications over thecommunication network 56 using one or more cellular communicationsprotocols, as are discussed herein. In various embodiments, the userdevice 54 includes a visual display, such as a touch-screen graphicaldisplay, or other display.

The remote transportation system 52 includes one or more backend serversystems, which may be cloud-based, network-based, or resident at theparticular campus or geographical location serviced by the remotetransportation system 52. The remote transportation system 52 can bemanned by a live advisor, or an automated advisor, or a combination ofboth. The remote transportation system 52 can communicate with the userdevices 54 and the autonomous vehicles 10 a-10 n to schedule rides,dispatch autonomous vehicles 10 a-10 n, and the like. In variousembodiments, the remote transportation system 52 stores accountinformation such as subscriber authentication information, vehicleidentifiers, profile records, behavioral patterns, and other pertinentsubscriber information.

In accordance with a typical use case workflow, a registered user of theremote transportation system 52 can create a ride request via the userdevice 54. The ride request will typically indicate the passenger'sdesired pickup location (or current GPS location), the desireddestination location (which may identify a predefined vehicle stopand/or a user-specified passenger destination), and a pickup time. Theremote transportation system 52 receives the ride request, processes therequest, and dispatches a selected one of the autonomous vehicles 10a-10 n (when and if one is available) to pick up the passenger at thedesignated pickup location and at the appropriate time. The remotetransportation system 52 can also generate and send a suitablyconfigured confirmation message or notification to the user device 54,to let the passenger know that a vehicle is on the way.

As can be appreciated, the subject matter disclosed herein providescertain enhanced features and functionality to what may be considered asa standard or baseline autonomous vehicle 10 and/or an autonomousvehicle based remote transportation system 52. To this end, anautonomous vehicle and autonomous vehicle based remote transportationsystem can be modified, enhanced, or otherwise supplemented to providethe additional features described in more detail below.

In accordance with various embodiments, the controller 34 implements anautonomous driving system (ADS) 70 as shown in FIG. 3. That is, suitablesoftware and/or hardware components of the controller 34 (e.g., theprocessor 44 and the computer-readable storage device 46) are utilizedto provide an autonomous driving system 70 that is used in conjunctionwith vehicle 10.

In various embodiments, the instructions of the autonomous drivingsystem 70 may be organized by function, module, or system. For example,as shown in FIG. 3, the autonomous driving system 70 can include acomputer vision system 74, a positioning system 76, a guidance system78, and a vehicle control system 80. As can be appreciated, in variousembodiments, the instructions may be organized into any number ofsystems (e.g., combined, further partitioned, etc.) as the disclosure isnot limited to the present examples.

In various embodiments, the computer vision system 74 synthesizes andprocesses sensor data and predicts the presence, location,classification, and/or path of objects and features of the environmentof the vehicle 10. In various embodiments, the computer vision system 74can incorporate information from multiple sensors, including but notlimited to cameras, lidars, radars, and/or any number of other types ofsensors.

The positioning system 76 processes sensor data along with other data todetermine a position (e.g., a local position relative to a map, an exactposition relative to lane of a road, vehicle heading, velocity, etc.) ofthe vehicle 10 relative to the environment. The guidance system 78processes sensor data along with other data to determine a path for thevehicle 10 to follow. The vehicle control system 80 generates controlsignals for controlling the vehicle 10 according to the determined path.

In various embodiments, the controller 34 implements machine learningtechniques to assist the functionality of the controller 34, such asfeature detection/classification, obstruction mitigation, routetraversal, mapping, sensor integration, ground-truth determination, andthe like.

As mentioned briefly above, all or parts of the perception system 100 ofFIG. 1 is included within the ADS 70, for example, as part of thecomputer vision system 74.

For example, as shown in more detail with regard to FIG. 4 and withcontinued reference to FIGS. 1-3, the perception system 100 includes animage uncertainty estimation module 110, an uncertain objectdetermination module 112, a lighting device position control module 114,a multi-sensor position control module 116, and an object recognitionmodule 118. As can be appreciated, any of the modules shown may becombined and/or further partitioned in various embodiments. In variousembodiments, inputs to the modules may be received from the sensorsystem 28, received from other controllers 34 (not shown) within thevehicle 10, and/or received from other modules within the controller 34of the vehicle 10.

In various embodiments, the image uncertainty estimation module 110receives as input image data 120. The image data 120 may be provided bya camera or other imaging device of the sensor system 28. The imageuncertainty estimation module 110 processes the image data 120 anddetermines any areas within the image where object detection isuncertain. For example, when low light conditions exist many pixels inthe resulting image may provide a solid, dark color. The imageuncertainty estimation module 110 produces uncertainty map data 122indicating uncertain areas within the image translated into a map orreal-world coordinates.

The uncertain object determination module 112 receives as input theuncertainty map data 122. The uncertain object determination module 112generates active sensor control signals 124 to control an active sensorto obtain active sensor data 126 from the environment within theidentified uncertain area. In various embodiments the active sensor is alow resolution lidar sensor of the sensor system 28.

The uncertain object determination module 112 processes the activesensor data 126 to determine if a potential object may be present withinthe uncertain area, and to prioritize or rank the uncertain areas basedon the location of the area and/or the potential presence of an object.For example, the uncertain object determination module 112 processes theactive sensor data 126 to determine if any of the data indicates thepresence of a potential object. In another example, the uncertain objectdetermination module 112 process the active sensor data 126 to determinea location of the uncertain areas relative to a current lane of travelof the vehicle 10. For example, uncertain areas may be identified as offroad, in the same lane, in opposing lane, off in a distance, etc. Theuncertain object determination module then uses the identified objectand/or location of the uncertain area to prioritize the uncertain areasand provide object location and priority data 128. For example, areaslocated in the same lane as the vehicle 10 and including a potentialobject may be given a highest priority, areas located in an opposinglane to the vehicle and identifying a potential object may be given anext highest priority, potential objects located off in a distance maybe given a next highest priority, and so on.

The lighting device position control module 114 receives as input thelocation and priority data 128. Based on the inputs, the lighting deviceposition control module 114 determines a position in the environment tofocus the lighting device 27. For example, the lighting device positioncontrol module 114 selects the uncertain area with the highest priorityand determines a position based on the location associated with theselected uncertain area. The lighting device position control module 114generates control signals 130 that control the actuator devicesassociated with the lighting device 27 to adjust the current position ofthe lighting device 27 to the determined position and/or to activate thelighting device 27 such that the uncertain area is illuminated.

The multi-sensor control module 116 receives as input the location andpriority data 128. Based on the inputs, the multi-sensor control module116 determines a position to place sensors such that the location of theuncertain area can be re-sensed, now that the area has been illuminated.In various embodiments, the sensors can include a camera and a highresolution lidar beam of the sensor system 28. For example, themulti-sensor control module 116 selects the uncertain area with thehighest priority and determines a position based on the locationassociated with the uncertain area. The multi-sensor position controlmodule 116 generates control signals 132, 134 that control the actuatordevices associated with the sensors to adjust the current position ofthe sensors to the determined positions and/or to activate the sensorssuch that the uncertain area is sensed.

The object recognition module 118 receives sensor data 136, 138 from thecontrolled sensors, for example the camera and the high resolutionlidar. The object recognition module 118 processes the sensor data 136,138 from the camera to determine contextual information associated withthe area. The object recognition module 118 processes the sensor datafrom the high resolution lidar to determine depth information associatedwith the area. The object recognition module 18 fuses the contextualinformation and the depth information to create an indication of anobject within the area. The fused information is then processed usingone or more machine learning methods to identify and/or classify theobject within the area. Any identified objects are then provided asobject data 140 to other control systems for controlling operation ofand navigating the vehicle 10.

Referring now to FIG. 5, and with continued reference to FIGS. 1-4, aflowchart illustrates a control method 200 that can be performed by theperception system 100 of FIG. 1 in accordance with the presentdisclosure. As can be appreciated in light of the disclosure, the orderof operation within the method is not limited to the sequentialexecution as illustrated in FIG. 5, but may be performed in one or morevarying orders as applicable and in accordance with the presentdisclosure. In various embodiments, the method 200 can be scheduled torun based on one or more predetermined events, and/or can runcontinuously during operation of the autonomous vehicle 10.

In one embodiment, the method may begin at 205. The image data 120 isobtained from, for example, the camera of the vehicle 10 at 210. Theimage data 120 is processed at 220 to identify any uncertain areas arebuild the uncertainty map data 122. If no uncertain areas are identifiedat 230, the method may end at 310. If, however, uncertain areas areidentified at 230, the active sensor data 126 is obtained from, forexample, a lidar sensor, for that area at 240. The active sensor data126 is processed at 250 to confirm the presence of any uncertain objectswithin the uncertain area, a location of the uncertain objects, and apriority of the uncertain object, for example, as discussed above toproduce the object location and priority data 128.

If no uncertain object exists or an uncertain object with low priorityexists, the method may end at 310. If, however, any uncertain objectsare identified, and the uncertain objects include a priority thatindicates a threat at 260, a position of the lighting device 27 isdetermined based on the location of the uncertain object and thelighting device 27 is controlled to the determine position such that itilluminates the area in the environment associated with the uncertainobject at 270. Similarly, positions of the sensors (e.g., the camera andthe high resolution lidar) are determined based on the location of theuncertain object and the sensors are controlled to the positions suchthat they sense the area in the environment associated with theuncertain object to obtain sensor data as the camera data 136 and thelidar data 138 at 280.

Thereafter, the sensor data is processed and fused, for example, asdiscussed above at 2900. The fused data is then evaluated to determineobject data 140 that may be used to control the vehicle 10 at 300.Thereafter, the method may end at 310.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or exemplary embodiments are only examples, and arenot intended to limit the scope, applicability, or configuration of thedisclosure in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the exemplary embodiment or exemplary embodiments. Itshould be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of thedisclosure as set forth in the appended claims and the legal equivalentsthereof.

What is claimed is:
 1. A method for detecting objects within anenvironment of a vehicle, comprising: receiving, by a processor, imagedata sensed from the environment of the vehicle; determining, by aprocessor, an area within the image data that object identification isuncertain; controlling, by the processor, a position of a lightingdevice to illuminate a location in the environment of the vehicle,wherein the location is associated with the area; controlling, by theprocessor, a position of one or more sensors to obtain sensor data fromthe location of the environment of the vehicle while the lighting deviceis illuminating the location; identifying, by the processor, one or moreobjects from the sensor data; and controlling, by the processor, thevehicle based on the one or more objects.
 2. The method of claim 1,wherein the one or more sensors comprise a camera and a lidar or ridar.3. The method of claim 1, wherein the position of the lighting deviceincludes at least one of a yaw, and a pitch relative to the vehicle. 4.The method of claim 1, wherein the position of the one or more sensorsincludes at least one of a yaw, a pitch, and a roll relative to thevehicle.
 5. The method of claim 1, further comprising determining apriority of the area, and wherein the controlling the position of thelighting device and the controlling the position of the one or moresensors is based on the priority of the area.
 6. The method of claim 5,wherein the determining the priority is based on lidar data sensed fromthe environment of the vehicle.
 7. The method of claim 6, wherein thedetermining the priority is based on a location of the area relative tolane of travel of the vehicle.
 8. The method of claim 6, wherein thedetermining the priority is based on a detected potential object in thearea.
 9. The method of claim 1, wherein the determining the area withinthe image that object detection is uncertain is based on a pixel valuefor threshold amount of pixels within the image.
 10. A computerimplemented system for detecting objects within an environment of avehicle, comprising: a computer readable medium configured to storeinstructions; and a processor configured to perform the instructions,the instructions when performed, receive image data sensed from theenvironment of the vehicle, determine an area within the image data thatobject identification is uncertain, control a position of a lightingdevice to illuminate a location in the environment of the vehicle,wherein the location is associated with the area, control a position ofone or more sensors to obtain sensor data from the location of theenvironment of the vehicle while the lighting device is illuminating thelocation, identify one or more objects from the sensor data, and controlthe vehicle based on the one or more objects.
 11. The system of claim10, wherein the one or more sensors comprise a camera and a lidar or aradar.
 12. The system of claim 10, wherein the position of the lightingdevice includes at least one of a yaw, and a pitch relative to thevehicle.
 13. The system of claim 10, wherein the position of the one ormore sensors includes at least one of a yaw, a pitch, and a rollrelative to the vehicle.
 14. The system of claim 10, wherein theinstructions determine a priority of the area, and wherein thecontrolling the position of the lighting device and the controlling theposition of the one or more sensors is based on the priority of thearea.
 15. The system of claim 14, wherein the instructions determine thepriority based on lidar data sensed from the environment of the vehicle.16. The system of claim 15, wherein the instructions determine thepriority based on a location of the area relative to lane of travel ofthe vehicle.
 17. The system of claim 16, wherein the instructionsdetermine the priority based on a detected potential object in the area.18. The system of claim 10, wherein the instructions determine the areawithin the image that object detection is uncertain is based on a pixelvalue for threshold amount of pixels within the image.
 19. An autonomousvehicle, comprising: a camera that senses image data from an environmentof the autonomous vehicle; at least one of a lidar and a radar thatsenses sensor data from the environment of the vehicle; a steerablelighting device that selectively illuminates the environment of thevehicle; and a controller configured to, by a processor, receive imagedata sensed from the environment of the vehicle, determine an areawithin the image data that object identification is uncertain, control aposition of the lighting device to illuminate a location in theenvironment of the vehicle, wherein the location is associated with thearea, control a position of the camera and the at least one of lidar andradar to obtain the image data and the sensor data from the location ofthe environment of the vehicle while the lighting device is illuminatingthe location, identify one or more objects from the sensor data, andcontrol the vehicle based on the one or more objects.
 20. The autonomousvehicle of claim 19, wherein the camera, the at least one of lidar andradar, and the lighting device are movably coupled to the vehicle, andwherein the position is relative to the vehicle.